Sentiment Analysis

Abstract

SA = algorithmically mapping the emotion or opinion of a text.

Find this repository: https://github.com/libjohn/workshop_textmining

Much of this review comes from the site: https://juliasilge.github.io/tidytext/

The primary library package tidytext enables all kinds of text mining. See Also this helpful free online book: Text Mining with R: A Tidy Approach by Silge and Robinson

Data

We’ll look at some books by Jane Austen, an 18th century novelist. Austen explored women and marriage within the British upper class. The novelist has a unique and well earned following within literature. Her works is consistently discussed and honored. To this day, Austen’s novels are the source of many adaptations, written and on-screen. Through the janeaustenr package we can access and mine the text of six Austen novels. We can call the collection of novels a corpra. An individual novel is a corpus.

Austen is best know for six published works:

Data Cleaning

Text mining typically requires a lot of data cleaning. In this case, we start with the janeaustenr collection that has already been cleaned. Nonetheless, further data wrangling is required. First, identifying a line number for each line of text in each book.

Identify line numbers

Tokens

To work with these data as a tidy dataset, we need to restructure the data through tokenization. In our case a token is a single word. We want one-token-per-row. The unnest_tokens() function (tidytext package) will convert a data frame with a text column into the one-token-per-row format.

Token
Tokenization
defined

The default tokenizing mode is “words”. With the unnest_tokens() function, tokens can be: words, characters, character_shingles, ngrams, skip_ngrams, sentences, lines, paragraphs, regex, tweets, and ptb (Penn Treebank).

Process

1. Group by line number (above)
2. Make each single word a token

Now that the data is in the one-word-per-row format, we can manipulate it with tidy tools like dplyr.

Stop Words

tidytext::get_stopwords()

Remove stop-words from the books.

Joining, by = "word"

Join types

Customize your dictionaries

You can customize stop-words data frames, sentiment data frames, etc.

There are various stop words dictionaries. Here we add the stop word, “farfegnugen” to a custom dictionary. If Jane Austen ever used the word “farfegnugen” that would be weird, or bad. So we will take pains to not calculate the sentiment of that word - whether or not the term shows up in a sentiment dictionary. That is, we will remove the word by making it a part of a customized stop-words dictionary.

[1] "snowball"      "stopwords-iso" "misc"          "smart"         "marimo"        "ancient"       "nltk"         
[8] "perseus"      
 [1] "da" "de" "en" "es" "fi" "fr" "hu" "ir" "it" "nl" "no" "pt" "ro" "ru" "sv"

Calculate word frequency

How many Austen countable words are there if we remove snowball stop-words? There are 14375 countable words.

Word clouds

Basic word cloud

A non-interactive word cloud.

Your Turn: Exercise 1

Goal: Make a basic word cloud for the novel, Pride and Predjudice, pride_prej_novel

1. Prepare

2. Tokenize pride_prej_novel with unnest_tokens()

3. Remove stop-words

4. calculate word frequency

5. make a simple wordcloud

Sentiment Analysis

get_sentiments()

Let’s see what positive words exist in the bing dictionary. Then, count the frequency of those positive words that exist in Emma.

Joining, by = "word"

Prepare to visualize sentiment score

Match all the Austen books to the bing sentiment dictionary. Count the word frequency.

Joining, by = "word"

Calculate sentiment

Algorithm: sentiment = positive - negative

Define a section of text.

"Small sections of text may not have enough words in them to get a good estimate of sentiment while really large sections can wash out narrative structure. For these books, using 80 lines works well, but this can vary depending on individual texts… – Text Mining with R

bing <- get_sentiments("bing")

janeaustensentiment <- tidy_books %>% 
  inner_join(bing) %>% 
  count(book, index = line %/% 80, sentiment) %>%                          # `%/%` = int division ; 80 lines / section
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>%    # spread(sentiment, n, fill = 0)
  mutate(sentiment = positive - negative)                                      # ALGO!!!

Joining, by = "word"

  
janeaustensentiment

Viz it

Preparation: Most common positive and negative words

Joining, by = "word"

Viz it too

Dictionaries

What other dictionaries are available? How to choose?

• Without Dictiionaries there is no sentiment analysis
• Sentiment Analysis: Analyzing Lexicon Quality and Estimation Errors
• Limits of the Bing, AFINN, and NRC Lexicons with the Tidytext Package in R
• Case Study with Harry Potter

Afinn

What words in Emma match the AFINN dictionary?

Joining, by = "word"
Joining, by = "word"

Make Sections

Just as we calculated sentiment, above, make sections of 80 words then calculate sentiment.

`summarise()` ungrouping output (override with `.groups` argument)

Viz it

Resources

• Tidytext package
• Book: Text Mining with R by Silge and Robinson
• Data Wrangling with dplyr: (video | workshop)
• Data Visualization with ggplot2: (video | workshop)

---

John Little
Rfun
Center for Data & Visualization Sciences

 

CC BY-NC
Creative Commons: Attribution, Non-commercial
https://creativecommons.org/licenses/by-nc/4.0/